Vehicle periphery image display device and camera adjustment method

ABSTRACT

An onboard periphery image display device for adjusting transformation rule of each camera mounted on an automobile is provided. A predetermined calibration pattern of a calibration member is placed in a capture area of a reference camera and another calibration pattern of the calibration member is placed in a capture area of an adjustment-targeted camera. The onboard periphery image display detects a coordinate of an image of a reference calibration pattern based on the image of the predetermined calibration pattern captured by the reference camera, detects an coordinate of an image of an adjustment calibration pattern based on the image of the another calibration pattern captured by the adjustment-targeted camera, and adjusts the transformation rule determined for the adjustment-targeted camera based on the detected coordinates.

CROSS REFERENCE TO RELATED APPLICATION

This application is based on Japanese Patent Applications No.2013-204474 filed on Sep. 30, 2013, the disclosure of which isincorporated herein by reference.

TECHNICAL FIELD

The present disclosure relates to a vehicle periphery image displaydevice to display an image around a vehicle and relates to an adjustmentmethod of a camera used for the vehicle periphery image display device.

BACKGROUND ART

A publicly known technology transforms an image captured by an onboardcamera into a projective image (bird's-eye image) representing thevehicle periphery (a state of the vehicle periphery viewed from abovethe vehicle).

A recently disclosed technology calibrates (adjusts) a camera when thecamera or an electronic controller to control the camera is replaced orwhen a camera position or posture is misaligned while in use.

In the disclosed technology, the camera uses a transformation rule(parameter) to transform an image captured by the camera into aprojective image. The transformation rule may come incorrect due toreplacement of the camera or misalignment of the camera position orposture. In such a case, the technology assumes the camera to be anadjustment-targeted camera and corrects the transformation rule.

The description below explains a method known as the technology tocorrect the transformation rule.

When adjusting the camera's transformation rule at a factory, aspecially shaped and sized calibration pattern is painted at apredetermined location of the ground in the factory. A vehicle is movedto a predetermined place to adjust the camera's transformation rule.This determines a positional relationship between the calibrationpattern and the vehicle and accordingly determines a positionalrelationship between the calibration pattern and the camera provided forthe vehicle. An arithmetic operation is used to adjust the camera'stransformation rule.

A car dealership may not be able to ensure a special place to adjust thecamera's transformation rule when adjusting the camera's transformationrule. To address this, the technology disclosed in patent literature 1places one calibration pattern in a partially overlapping range betweena capture range of the adjustment-targeted camera and a capture range ofan adjacent reference camera (no adjustment needed). The technologyadjusts the transformation rule determined for the adjustment-targetedcamera so that the calibration pattern matches coordinates of thecalibration pattern in the video in an image space that isprojection-transformed according to a transformation rule determined forthe reference camera.

In adjusting the camera's transformation rule by placing the calibrationpattern at arbitrary location in the car dealership, the capture rangeof the adjustment-targeted camera overlaps with the capture range of acamera serving as a candidate for the reference camera and thecalibration pattern is placed in the overlapping range. Otherwise,adjusting the transformation rule determined for the adjustment-targetedcamera is difficult.

Additionally, depending on positions to place the calibration pattern,accuracy of the captured image of the calibration pattern is low,disabling accurate adjustment.

PRIOR ART LITERATURES Patent Literature

Patent Literature 1: JP-2010-244326A

SUMMARY OF INVENTION

The present disclosure has been made in consideration of the foregoing.It is an object of the disclosure to provide a vehicle periphery imagedisplay device and a camera adjustment method capable of easilyadjusting a transformation rule of an adjustment-targeted camera evenwhen a calibration pattern is placed in a range except an overlapbetween a capture range of the adjustment-targeted camera and a capturerange of a reference camera.

An onboard periphery image display device in a first example of thepresent disclosure is mounted on an automobile and comprises a pluralityof cameras, a vehicle periphery image generation portion, a displayapparatus, and an adjustment portion. The plurality of cameras aremounted on an automobile to image a periphery of the automobile. Inaccordance with a transformation rule determined for each cameraprovided to capture the image, the vehicle periphery image generationportion applies projection transform to the images captured by thecameras to generate a vehicle periphery image synthesized in a singleimage space. The vehicle periphery image represents the periphery of theautomobile observed from a specified viewpoint. The display apparatusdisplays the vehicle periphery image generated by the vehicle peripheryimage generation portion. The adjustment portion adjusts thetransformation rule determined for each of the cameras. In a situationwhere: a calibration member, in which a plurality of calibrationpatterns with predetermined sizes and shapes have a predeterminedpositional relationship, is placed around the automobile; apredetermined calibration pattern of the calibration patterns is placedin a capture area of a reference camera, which is a camera serving as areference to adjust the transformation rule; and another calibrationpattern of the calibration patterns is placed in a capture area of anadjustment-targeted camera, which is a camera targeted for adjustment ofthe transformation rule, the adjustment portion: detects a coordinate ofan image of a reference calibration pattern through applying theprojection transform to the image of the predetermined calibrationpattern of the calibration member captured by the reference camera inaccordance with the transformation rule determined for the referencecamera; detects an coordinate of an image of an adjustment calibrationpattern through applying the projection transform to the image of theanother calibration pattern of the calibration member captured by theadjustment-targeted camera in accordance with the transformation ruledetermined for the adjustment-targeted camera; performs detecting acoordinate of an image of a provisional reference calibration patterncorresponding to the reference calibration pattern based on a positionalrelationship of the coordinate of the image of the adjustmentcalibration pattern with the calibration patterns or detecting acoordinate of an image of a provisional adjustment calibration patterncorresponding to the adjustment calibration pattern based on apositional relationship of the coordinate of the image of the referencecalibration pattern with the calibration patterns; and adjusts thetransformation rule determined for the adjustment-targeted camera sothat the coordinate of the image of the provisional referencecalibration pattern matches the coordinate of the image of the referencecalibration pattern or the coordinate of the image of the provisionaladjustment calibration pattern matches the coordinate of the image ofthe adjustment calibration pattern.

A camera adjustment method in a first example of the present disclosureis provided for an onboard periphery image display device including aplurality of cameras that are mounted on an automobile to image aperiphery of the automobile; a vehicle periphery image generationportion that, in accordance with a transformation rule determined foreach camera provided to capture the image, applies projection transformto the images captured by the cameras to generate a vehicle peripheryimage synthesized in a single image space, wherein the vehicle peripheryimage represents the periphery of the automobile observed from aspecified viewpoint; and a display apparatus that displays the vehicleperiphery image generated by the vehicle periphery image generationportion, and the camera adjustment method is provided for adjusting thetransformation rules for the cameras. In a situation where: acalibration member, in which a plurality of calibration patterns withpredetermined sizes and shapes have a predetermined positionalrelationship, is placed around the automobile; a predeterminedcalibration pattern of the calibration patterns is placed in a capturearea of a reference camera, which is a camera serving as a reference toadjust the transformation rule; and another calibration pattern of thecalibration patterns is placed in a capture area of anadjustment-targeted camera, which is a camera targeted for adjustment ofthe transformation rule, the camera adjustment method comprisesdetecting a coordinate of an image of a reference calibration patternthrough applying the projection transform to the image of thepredetermined calibration pattern of the calibration member captured bythe reference camera in accordance with the transformation ruledetermined for the reference camera; detecting an coordinate of an imageof an adjustment calibration pattern through applying the projectiontransform to the image of the another calibration pattern of thecalibration member captured by the adjustment-targeted camera inaccordance with the transformation rule determined for theadjustment-targeted camera; performing detecting a coordinate of animage of a provisional reference calibration pattern corresponding tothe reference calibration pattern based on a positional relationship ofthe coordinate of the image of the adjustment calibration pattern withthe calibration patterns or detecting a coordinate of an image of aprovisional adjustment calibration pattern corresponding to theadjustment calibration pattern based on a positional relationship of thecoordinate of the image of the reference calibration pattern with thecalibration patterns; and adjusting the transformation rule determinedfor the adjustment-targeted camera so that the coordinate of the imageof the provisional reference calibration pattern matches the coordinateof the image of the reference calibration pattern or the coordinate ofthe image of the provisional adjustment calibration pattern matches thecoordinate of the image of the adjustment calibration pattern.

Even if capture ranges for the cameras do not overlap with each other,the onboard periphery image display device and the camera adjustmentmethod according to the present disclosure can easily calibrate theadjustment-targeted camera by placing the calibration patterns in thecorresponding capture ranges and performing the above-mentioned process.

In a conventional one, a very small (single) pattern may be displayed ona screen. Even in such a case, the onboard periphery image displaydevice and the camera adjustment method according to the presentdisclosure uses the calibration member including different patternsplaced in capture ranges for the different cameras to improve theaccuracy for the cameras to detect calibration patterns. The onboardperiphery image display device and the camera adjustment methodaccording to the present disclosure can advantageously accuratelyperform the calibration by performing the above-mentioned process.

An onboard periphery image display device in a second example of thepresent disclosure is mounted on an automobile and comprises a pluralityof cameras, a vehicle periphery image generation portion, a displayapparatus and an adjustment portion. The plurality of cameras aremounted on an automobile to image a periphery of the automobile. Inaccordance with a transformation rule determined for each cameraprovided to capture the image, the vehicle periphery image generationportion applies projection transform to the images captured by thecameras to generate a vehicle periphery image synthesized in a singleimage space. The vehicle periphery image represents the periphery of theautomobile observed from a specified viewpoint. The display apparatusdisplays the vehicle periphery image generated by the vehicle peripheryimage generation portion. The adjustment portion adjusts thetransformation rule determined for each of the cameras. In a situationwhere: calibration members, in which a plurality of calibration patternswith predetermined sizes and shapes and having a predeterminedpositional relationship, are placed around the automobile; apredetermined calibration pattern of the calibration patterns is placedin a capture area of a predetermined camera; and another calibrationpattern of the calibration patterns is placed in a capture area ofanother camera,

the adjustment portion: of one set of two adjustment-targeted camerasthat are capable of independently capturing two calibration patterns ofa predetermined calibration member of the calibration members, uses onecamera as a provisional reference camera and the other camera as anadjustment-targeted camera; detects a coordinate of an image of aprovisional reference calibration pattern (B) through, in accordancewith the transformation rule determined for the provisional referencecamera, applying the projection transform to the image of apredetermined calibration pattern of the two calibration patternscaptured by the provisional reference camera; detects a coordinate of animage of an adjustment calibration pattern through, in accordance withthe transformation rule determined for the adjustment-targeted camera,applying the projection transform to the image of the other calibrationpattern of the two calibration patterns captured by theadjustment-targeted camera; performs detecting a coordinate of an imageof a provisional reference calibration pattern (B′) corresponding to theprovisional reference calibration pattern (B) based on a positionalrelationship of the coordinate of the image of the adjustmentcalibration pattern with the two calibration patterns or detecting acoordinate of an image of a provisional adjustment calibration patterncorresponding to the adjustment calibration pattern based on apositional relationship of the coordinate of the image of theprovisional reference calibration pattern with the two calibrationpatterns; and adjusts the transformation rule determined for theadjustment-targeted camera so that the coordinate of the image of theprovisional reference calibration pattern (B′) matches the coordinate ofthe image of the provisional reference calibration pattern (B) or thecoordinate of the image of the provisional adjustment calibrationpattern matches the coordinate of the image of the adjustmentcalibration pattern. Said set-by-set-basis adjustment is applied toadjust the transformation rules for all the cameras.

A camera adjustment method in a second example of the present disclosureis provided for an onboard periphery image display device including: aplurality of cameras that are mounted on an automobile to image aperiphery of the automobile; a vehicle periphery image generationportion that, in accordance with a transformation rule determined foreach camera provided to capture the image, applies projection transformto the images captured by the cameras to generate a vehicle peripheryimage synthesized in a single image space, wherein the vehicle peripheryimage represents the periphery of the automobile observed from aspecified viewpoint; and a display apparatus that displays the vehicleperiphery image generated by the vehicle periphery image generationportion. The camera adjustment method is provided for adjusting thetransformation rules for the cameras. In a situation where: calibrationmembers, in which a plurality of calibration patterns with predeterminedsizes and shapes have a predetermined positional relationship, areplaced around the automobile; a predetermined calibration pattern of thecalibration patterns is placed in a capture area of a predeterminedcamera; and another calibration pattern of the calibration patterns isplaced in a capture area of another camera, the camera adjustment methodcomprises: of one set of two adjustment-targeted cameras that arecapable of independently capturing two calibration patterns of apredetermined calibration member of the calibration members, using onecamera as a provisional reference camera and the other camera as anadjustment-targeted camera; detecting a coordinate of an image of aprovisional reference calibration pattern (B) through, in accordancewith the transformation rule determined for the provisional referencecamera, applying the projection transform to the image of apredetermined calibration pattern of the two calibration patternscaptured by the provisional reference camera; detecting a coordinate ofan image of an adjustment calibration pattern through, in accordancewith the transformation rule determined for the adjustment-targetedcamera, applying the projection transform to the image of the othercalibration pattern of the two calibration patterns captured by theadjustment-targeted camera; performing detecting a coordinate of animage of a provisional reference calibration pattern (B′) correspondingto the provisional reference calibration pattern (B) based on apositional relationship of the coordinate of the image of the adjustmentcalibration pattern with the two calibration patterns or detecting acoordinate of an image of a provisional adjustment calibration patterncorresponding to the adjustment calibration pattern based on apositional relationship of the coordinate of the image of theprovisional reference calibration pattern with the two calibrationpatterns; and adjusting the transformation rule determined for theadjustment-targeted camera so that the coordinate of the image of theprovisional reference calibration pattern (B′) matches the coordinate ofthe image of the provisional reference calibration pattern (B) or thecoordinate of the image of the provisional adjustment calibrationpattern matches the coordinate of the image of the adjustmentcalibration pattern. Said set-by-set-basis adjustment is applied toadjust the transformation rules for all the cameras.

In the onboard periphery image display device and the camera adjustmentmethod according to the present disclosure, even when all cameras are tobe adjusted due to, for example, replacement of the electronic controlunit, it is possible to easily calibrate adjustment-targeted cameras byplacing the calibration patterns in the corresponding capture ranges andperforming the above-mentioned process even if the cameras' captureranges do not overlap.

In a conventional one, a very small single pattern may be displayed on ascreen. In such a case, the onboard periphery image display device andthe camera adjustment method according to the present disclosure usesthe calibration member containing different calibration patterns placedin capture ranges for the different cameras to improve the accuracy forthe cameras to detect calibration patterns. The onboard periphery imagedisplay device and the camera adjustment method can advantageouslyaccurately perform the calibration by performing the above-mentionedprocess.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram illustrating a configuration of a vehicleperiphery image display device according to a first embodiment;

FIG. 2 is a plan view illustrating a calibration sheet includingpatterns A and B;

FIG. 3 is an explanatory diagram illustrating placement of a calibrationsheet (viewed from above a vehicle) according to the first embodiment;

FIG. 4 is a flowchart illustrating a camera calibration method accordingto the first embodiment;

FIG. 5A is an explanatory diagram illustrating distortion correctionapplied to an image captured by a wide angle camera;

FIG. 5B is a line drawing of FIG. 5A;

FIG. 6 is an explanatory diagram illustrating an example of a vehiclecoordinate system and camera parameters;

FIG. 7 is a flowchart illustrating a method of updating Roll, Pitch, andZ belonging to the calibration method according to the first embodiment;

FIG. 8 is an explanatory diagram illustrating placement of a calibrationsheet (viewed from above a vehicle) according to a second embodiment;

FIG. 9 is a flowchart illustrating a camera calibration method accordingto the second embodiment;

FIG. 10 is an explanatory diagram illustrating placement of acalibration sheet (viewed from above a vehicle) according to a thirdembodiment;

FIG. 11 is an explanatory diagram illustrating placement of acalibration sheet (viewed from above a vehicle) according to a fourthembodiment;

FIG. 12A illustrates an image resulting from applying distortioncorrection to an image captured by a left camera according to the fourthembodiment;

FIG. 12B illustrates an image resulting from enlarging an areacorresponding to pattern A in FIG. 12A;

FIG. 12C illustrates an image resulting from enlarging an areacorresponding to pattern B in FIG. 12A;

FIG. 12D is a line drawing of FIG. 12A;

FIG. 12E is a line drawing of FIG. 12B;

FIG. 12F is a line drawing of FIG. 12C;

FIG. 13 is an explanatory diagram illustrating an example of placementof a calibration sheet (viewed from above a vehicle) according to afifth embodiment;

FIG. 14 is an explanatory diagram illustrating placement of acalibration sheet (viewed from above a vehicle) according to a sixthembodiment;

FIG. 15A is a flowchart illustrating the first half of the cameracalibration method according to the sixth embodiment;

FIG. 15B is a flowchart illustrating the second half of the cameracalibration method according to the sixth embodiment;

FIG. 16 is an explanatory diagram illustrating relationship between aprovisional reference camera coordinate system and a vehicle coordinatesystem (viewed from above a vehicle);

FIG. 17A is an explanatory diagram illustrating a method oftransformation to a vehicle coordinate system and a technique of findingθ1 (viewed from above a vehicle);

FIG. 17B is an explanatory diagram illustrating a method oftransformation to a vehicle coordinate system and a technique of findingθ2 (viewed from above a vehicle);

FIG. 18A is an explanatory diagram illustrating a method oftransformation to a vehicle coordinate system and a technique ofrotating the coordinate around the origin of a provisional referencecamera coordinate system (viewed from above a vehicle);

FIG. 18B is an explanatory diagram illustrating a method oftransformation to a vehicle coordinate system and a technique of findingOFFSET_X and OFFSET_Y (viewed from above a vehicle);

FIG. 19A is an explanatory diagram illustrating a technique ofestimating pattern B using pattern A; and

FIG. 19B is an explanatory diagram illustrating a technique ofestimating pattern B using patterns A and C according to a seventhembodiment.

EMBODIMENTS FOR CARRYING OUT INVENTION

The description below explains embodiments of a vehicle periphery imagedisplay device 1 according to the present disclosure with reference tothe accompanying drawings. The vehicle periphery image display device 1is provided as an example of onboard periphery image display devices.

(First Embodiment)

The vehicle periphery image display device 1 according to an embodimentcalibrates (adjusts) a camera by adjusting a transformation rule ofprojection transform applied to an image captured by a camera (onboardcamera) mounted on a vehicle.

a) The description below first explains a basic configuration of thevehicle periphery image display device 1 according to the embodimentwith reference to FIG. 1.

As illustrated in FIG. 1, the vehicle periphery image display device 1according to the embodiment is mounted on a vehicle (automobile) anddisplays a captured image around the vehicle. The vehicle peripheryimage display device 1 includes an electronic control unit (ECU) 3, afront camera 5, a left camera 7, a right camera 9, a rear camera 11, anda display apparatus 13. The ECU 3 includes a publicly knownmicrocomputer. The front camera 5 is placed at the front of the vehicle.The left camera 7 is placed at the left of the vehicle. The right camera9 is placed at the right of the vehicle. The rear camera 11 is placed atthe rear of the vehicle. The display apparatus 13 displays images.

A wide angle camera is used for each of the four cameras 5 through 11.The purpose is to transform images captured by the four cameras 5through 11 into a projective image (a bird's-eye image of the vehicleviewed from above the vehicle) synthesized in a single image space sothat the image covers an entire area around the vehicle.

The front camera 5 is embedded at the center of the vehicle front in thevehicle width direction so as to capture images of the front side of thevehicle.

The left camera 7 is embedded at the center or toward the front of theleft side of the vehicle in the longer direction of the vehicle so as tocapture images of the left side of the vehicle.

The right camera 9 is embedded at the center or toward the front of theright side of the vehicle in the longer direction of the vehicle so asto capture images of the right side of the vehicle.

The rear camera 11 is embedded at the center of the vehicle rear in thevehicle width direction so as to capture images of the back side of thevehicle.

The ECU 3 includes a power supply 15, an image input signal processingportion 17, an input signal processing portion 19, an image processingportion 21, memory 23, and an image output signal processing portion 25.The power supply 15 supplies the power to the ECU 3.

The image input signal processing portion 17 separates a video signalfrom analog capture data supplied from the cameras 5 through 11 andoutputs the video signal to the image processing portion 21.

The input signal processing portion 19 processes input shift positioninformation, vehicle speed information, steering angle information, anda signal from a diagnostic tool 27. The input signal processing portion19 supplies the image processing portion 21 with the shift positioninformation indicating forward or backward movement, the vehicle speedinformation indicating a low or high speed, the steering angleinformation indicating right or left rotation of a steering wheel or arotation angle, and an instruction from the diagnostic tool.

The image processing portion 21 converts a video signal from an analogsignal to a digital signal. The image processing portion 21 processesnecessary information using the memory 23 and retrieves necessary datafrom the below-described image process concerning a calibration sheet 31(see FIG. 2). The image processing portion 21 outputs the data (digitalsignal) to the image output signal processing portion 25.

As will be described later, the image process concerning the calibrationsheet 31 detects calibration pattern A (hereinafter simply referred toas a pattern) at one end of the calibration sheet 31 and detects patternB (maintaining predetermined positional relationship with pattern A)from the data of pattern A.

Based on the information from the input signal processing portion 19,the image processing portion 21 converts a video signal to a top-viewvideo when the vehicle speed is low, for example. In more detail, theimage processing portion 21 applies projection transform to imagescaptured by the cameras. The projection transform complies with atransformation rule predetermined for each of the cameras that capturedthe images. The image processing portion 21 synthesizes the images on asingle image space and generates a vehicle periphery image representingthe vehicle periphery observed from a predetermined viewpoint. The imageprocessing portion 21 is provided as an example of a vehicle peripheryimage generation portion and a vehicle periphery image generation means.

The memory 23 saves a video signal from the image processing portion 21and coordinate information about apex CT (see FIG. 2) as a feature pointfor pattern A or B.

The image output signal processing portion 25 converts a video signalinput from the image processing portion 21 from a digital signal to ananalog signal and outputs the video signal to the display apparatus 13.

The display apparatus 13 displays a video signal (consequently an image)input from the image output signal processing portion 25 and isavailable as a liquid crystal display, for example.

For example, a microcomputer process implements the input signalprocessing portion 19 or the image processing portion 21 mentionedabove.

b) The description below explains the calibration sheet 31 used forcamera calibration according to the first embodiment.

<Configuration of the Calibration Sheet 31>

The description below explains a configuration of the calibration sheet31.

As illustrated in FIG. 2, the calibration sheet 31 is provided as afoldable sheet 4700 mm×900 mm×5 mm thick. Square patterns A and B(equally shaped) are formed at both ends of the calibration sheet 31 inthe longer direction. Each pattern has an outside diameter of 900 mm.

It is preferable that a material for the calibration sheet 31 hardlycontract and expand with temperature or humidity (so as not to affectthe calibration). Specifically, the material is preferably cloth made ofhardly expansible and contractable chemical fiber.

Patterns A and B mentioned above are colored by printing or the like onthe surface or the like of the cloth used for the calibration sheet 31.Patterns A and B are provided with white square calibration marks 33 and35, in each which a 500-mm square is placed at the center of a blacksquare (base mark), for example. In other words, black square frames 34and 36 are formed to surround the white square calibration marks 33 and35.

The black-white contrast enables a publicly known image process todetect the four apexes CT (edges) of the calibration mark 33, 35 basedon a luminance difference in the image data of the pattern A, B. Inconsideration of the contrast, for example, a red color is used for anintermediate part between patterns A and B. While there have beendescribed the frames 34 and 36 in black and the intermediate part 37 inred, both may be provided in the same color as black or red.

Rod-like weights (end weights) 39 and 41, each 950 mm long, are providedat the ends of the calibration sheet 31 in the axial direction, namely,along the outer sides (shorter sides) in the axial direction of patternsA and B.

The end weights 39 and 41 prevent misalignment of the calibration sheet31 when placed. For this purpose, the end weights 39 and 41 may beadvantageously made of resin (e.g., ABS) or metal (e.g., iron) heavyenough to resist an effect of wind.

The calibration sheet 31 not only may be provided as a sheet but alsomay be painted on a road if patterns A and B conform to a specifiedsize.

Patterns A and B just need to be used to find apexes CT of thecalibration marks 33 and 35. Patterns A and B are not limited to besquare but may be formed to be polygonal such as a rectangle or atriangle.

<Placement of the Calibration Sheet 31>

The description below explains placement of the calibration sheet 31.

As illustrated in FIG. 3, the front camera 5 covers capture range SF (ahorizontally long rectangular range indicated by the broken line in thedrawing) in front of the vehicle. The left camera 7 covers capture rangeSL (a vertically long rectangular range indicated by the broken line inthe drawing) to the left side of the vehicle. Capture ranges SF and SLpartially overlap with each other (at the top left of the drawing). Thedescription below explains a case of placing the calibration sheet 31 soas to extend across capture ranges SF and SL. An overlap between captureranges SF and SL is referred to as overlapping range SFL.

In more detail, pattern A at one end of the calibration sheet 31 isplaced in capture range SF (except overlapping range SFL). Pattern B atthe other end of the calibration sheet 31 is placed in capture range SL(except overlapping range SFL).

c) The description below explains a calibration method according to thefirst embodiment.

For example, a signal from the diagnostic tool 27 determines whichcamera serves as a reference camera (that is correctly positioned ororiented and requires no calibration) and which camera serves as anadjustment-targeted camera (that requires calibration because the camerais incorrectly positioned or oriented or is replaced). The adjustmentaims at adjusting a transformation rule (parameter) for the camera'sprojection transform. The description below is directed to an example inwhich the left camera 7 serves as the reference camera and the frontcamera 5 serves as the adjustment-targeted camera.

For example, a signal from the diagnostic tool 27 starts thecalibration. The ECU 3 (e.g., image processing portion 21) correspondingto an example of an adjustment portion and an adjustment means receivesthe signal from the diagnostic tool 27 and executes a calibrationprocessing program to adjust the transformation rule as described below.

As illustrated in FIG. 4, the signal from the diagnostic tool 27 startsthe calibration processing program. At S100, the ECU 3 acquiresinformation about the reference camera and the adjustment-targetedcamera.

At S110, the front camera 5 serving as the adjustment-targeted cameracaptures an image. This allows the ECU 3 to acquire image data ofcapture range SF of the front camera 5. The image data contains patternA as a calibration pattern. The capture data is input to the image inputsignal processing portion 17.

At S120, the ECU 3 applies an image process to the image of the capturerange SF, the image being captured by the front camera 5. The ECU 3detects pattern A (in detail, pattern A as an image corresponding toactual pattern A) as described below.

This image process generates an image of an adjustment calibrationpattern GA. Actual patterns GA and GB are used to refer to the imagedata of each pattern A and B. The same applies to the other patterns.

The image input signal processing portion 17 separates a video signal(containing the information about luminance) from the input analogsignal. The image processing portion 21 converts the video signal fromthe analog signal to a digital signal and then detects pattern A.

Specifically, the ECU 3 uses a digitized video signal to apply publiclyknown distortion correction to a captured image.

An image is captured by a wide-angle lens and is therefore distorted(see upper parts of FIGS. 5A and 5B). For example, a straight line isdisplayed as if it were curved. The distortion needs to be corrected sothat the straight line is corrected and is displayed straight (see lowerparts of FIGS. 5A and 5B). The distortion correction is publicly knownand a description is omitted for simplicity.

After the distortion correction, the ECU 3 compares the luminance in thevideo signal and detects edges corresponding to apexes CT of pattern Abased on a luminance difference (an intersection between straightlines). Of the detected edges, the ECU 3 finds edges corresponding tothe edges (corners) of the white graphic (calibration mark 35) enclosedby the black frame area inside pattern A. The ECU 3 detects the edges asthe four apexes CT corresponding to pattern A. Subsequently, the memory23 stores the result of detecting the four apexes CT.

At this stage, only the distortion correction is applied to the image.Normally, pattern A is not represented as a square, but as aquadrilateral such as a trapezoid.

At S130, the image processing portion 21 retrieves (computes) andcorrects a parameter serving as the transformation rule for the frontcamera 5 based on the image data of pattern A stored in the memory 23.

Specifically, the front camera 5 uses parameters X, Y, Z, Roll, Pitch,and Yaw. The ECU 3 retrieves (computes) and corrects Roll, Pitch, and Zout of parameters so that pattern A conforms to a 500-mm square in termsof the shape and the size.

Determining Roll and Pitch determines the graphic shape. Determining Zdetermines the graphic size.

FIG. 6 illustrates a vehicle coordinate system, namely, a coordinatesystem based on the reference (origin) corresponding to a point on theground immediately below the front center of the vehicle. X denotes adistance in the X-axis direction or the front-back direction of thevehicle while the + side denotes forward. Y denotes a distance in theY-axis direction or the right-left direction of the vehicle while the +side denotes left. Z denotes a distance in the Z-axis direction or thevertical direction while the + side denotes above. For example, the leftcamera 7 uses parameters in the vehicle coordinate system as illustratedin FIG. 6.

The description below explains a method of computing Roll, Pitch, and Zbased on FIG. 7.

At S200 in FIG. 7, the ECU 3 updates Roll, Pitch, and Z.

For example, the ECU 3 divides the range corresponding to each parameterinto minute ranges (minute values) and sequentially updates the minutevalues of the parameter to update the parameter.

For example, suppose that Roll covers the range between −3 and +3degrees around the reference value. Then, the ECU 3 sequentially changesthe Roll value in increments of 0.1 degrees. The same applies to Pitchand Z.

At S210, the ECU 3 finds a projective image (bird's-eye image as a topview) from the image data using the parameters. Parameters Roll, Pitch,Z are sequentially changed. Parameters X, Y, and Yaw remain unchanged.The method of finding a projective image using the parameters ispublicly known and a description is omitted for simplicity.

At S220, the ECU 3 determines whether or not pattern A (in detail, thecalibration mark 35) in the projective image approximates to a square.Specifically, the ECU 3 finds a graphic most approximate to a square (animage most approximate to the image of the 500-mm square) fromcoordinates for the four apexes CT of the calibration mark 35 insidepattern A.

For example, the ECU 3 finds coordinates for apexes of a correct squareand coordinates for apexes of the quadrilateral in the projective image.The ECU 3 equates the graphic most approximate to a square with agraphic that minimizes the total value of distances between thecorresponding apexes. This method is publicly known as described inJapanese Patent No. 4555876 (see FIG. 7).

The ECU 3 repeats the process from S200 to S220 while updating theparameter values described above to find a graphic most approximate to asquare. The ECU 3 identifies the parameters used to find the graphicwith correct values for parameters Roll, Pitch, and Z for the frontcamera 5 and stores the parameter values in the memory 23.

The ECU 3 causes the memory 23 to store the coordinates for apexes CT ofthe calibration mark 35 in the graphic that is found as described abovefor pattern A and most approximates to a square.

The description returns to FIG. 4. At S130, the ECU 3 finds coordinatesfor apexes CT of the calibration mark 33 for provisional pattern B′,from the coordinates stored in memory 23 for apexes CT of thecalibration mark 35 in the graphic that are most approximate to a squarefor pattern A.

Because the dimensions of the calibration sheet 31 are pre-known, theECU 3 can find coordinates for apexes CT of provisional pattern B′ fromthe coordinates for apexes CT of pattern A and the dimensions of thecalibration sheet 31.

In the description below, an apostrophe “'” follows the letter of aprovisional pattern computed as a supplementary one.

At S150, the ECU 3 captures pattern B using the left camera 7 serving asthe reference camera, for which the adjustment of the transformationrule is not needed.

At S160, the capture data for pattern B is input to the image inputsignal processing portion 17. The image processing portion 21 thendetects pattern B in the same manner as detecting the pattern A. Theimage acquired by this image process is the reference calibrationpattern GB corresponding to actual pattern B as described above.

Specifically, because pattern B is square, the ECU 3 detects four apexesCT of the square for the calibration mark 33.

The image data acquired by the reference camera is transformed into aprojective image (by a correct transformation rule). Coordinate data forthe projective image is coordinate data in the vehicle coordinatesystem.

At S170, the ECU 3 approximates provisional pattern B′ to pattern Busing image data for provisional pattern B′ and pattern B. Namely, theECU 3 aligns apexes CT of the calibration mark for provisional patternB′ to those for pattern B so that the apexes CT for provisional patternB′ approximate to the corresponding apexes CT for pattern B. The ECU 3retrieves (computes) and corrects parameters for the front camera 5 soas to minimize a positional error.

The ECU 3 retrieves (computes) and corrects Yaw, X, and Y out ofparameters X, Y, Z, Roll, Pitch, and Yaw so as to minimize a positionalerror (i.e., the total of distances between the corresponding apexes CT)by adjusting the orientation and the position of provisional pattern B′.

The method of computing Yaw, X, and Y is similar to the method ofcomputing Z, Roll, and Pitch illustrated in FIG. 7.

Determining Yaw, X, and Y can determine the position and the rotationangle of a graphic on the plane.

The method of minimizing a positional error between two correspondinggraphics is publicly known and a description is omitted for simplicity.The method is detailed in Japanese Patent No. 4555876, for example.

At S180, the parameters X, Y, Z, Roll, Pitch, and Yaw after theretrieval (computation) for the front camera 5 are stored in the memory23 by the ECU 3. Then the processing is ended.

This enables to confirm that the adjustment using the parameters afteradjusting the front camera 5 has been correctly made.

Specifically, images captured by the front camera 5, the left camera 7,the right camera 9, and the rear camera 11 are first input to the imageinput signal processing portion 17. The image processing portion 21 thentransforms the images captured by the four cameras into a projectiveimage and combines them in a single image space (a space in the vehiclecoordinate system). The image output signal processing portion 25converts the projective image data into an analog signal and outputs thesignal. The display apparatus 13 displays the signal so that the imagecan be confirmed.

The memory 23 stores necessary information during processes in the ECU3. The image processing portion 21 sequentially processes theinformation using the memory 23.

A shift position, a vehicle speed, and a steering angle input to theinput signal processing portion 19 can determine from which viewpointthe space should be displayed on the screen. When the vehicle travels ata low speed, for example, a top view screen is selected to provide abird's-eye image so that the periphery of the vehicle can be viewed fromabove the vehicle.

d) Effects of the embodiment will be described.

The first embodiment uses the calibration sheet 31 including twopatterns A and B. Pattern A is placed in capture range SF of the frontcamera 5 serving as an adjustment-targeted camera. Pattern B is placedin capture range SL of the left camera 7 serving as a reference camera.Patterns A and B are not placed in overlapping range SFL between thecameras 5 and 7.

In this state, by performing the above-mentioned calibration, it ispossible to adjust the front camera 5 serving as an adjustment-targetedcamera.

For example, suppose that patters A and B are provided as calibrationpatterns based on the positional relationship in FIG. 3 and that thereference camera is the left camera 7 and the adjustment-targeted camerais the front camera 5. In this case, the left camera 7 serving as thereference camera can capture pattern B. However, the front camera 5serving as the adjustment-targeted camera cannot capture pattern B.

However, the front camera 5 can capture pattern A. The positionalrelationship between patterns A and B is already known. Pattern A can beused to compute provisional pattern B′ as an estimated position forpattern B. Establishing a correspondence between coordinates forprovisional pattern B′ and coordinates for provisional pattern B can setthe parameters or change the transformation rule for the front camera 5.

Even if capture ranges for the cameras 5 and 7 do not overlap with eachother, the first embodiment can easily calibrate the front camera 5serving as an adjustment-targeted camera by placing patterns A and B inthe corresponding capture ranges SF and SL and performing theabove-mentioned process.

Even if a screen displays (single) pattern A (or B) to be very small asis the case in a conventional one, the first embodiment uses thecalibration sheet 31 containing different patterns A and B placed incapture ranges SF and SL for the different cameras 5 and 7 to improvethe accuracy for the cameras 5 and 7 to detect patterns A and B. Thefirst embodiment can provide a technical advantage of accuratelyperforming the calibration by performing the above-mentioned process.

According to the first embodiment, the calibration sheet 31 is providedas an elongated (rectangular) member and patterns A and B are formed atboth ends in the longer direction. Patterns A and B can be easily placedin the corresponding capture ranges even if the capture ranges for thecameras are distant from each other.

According to the present embodiment, the calibration sheet 31 is made ofcloth (fiber) as a material that hardly expands and contracts withtemperature and humidity. The accurate calibration is always availableeven if the environment changes.

The calibration sheet 31 can be folded like a scroll, showing anadvantage of space-saving and excellent usability.

The end weights 39 and 41 as rod-like weights are provided at both endsof the calibration sheet 31 in the longer direction (outside patterns Aand B), enabling the calibration sheet 31 to be placed stably.

Materials of the weight may include resin (e.g., ABS) or metal (e.g.,iron).

(Second Embodiment)

The description below explains a second embodiment, in which anexplanation on contents similar to those in the first embodiment areomitted or simplified. The second embodiment uses the same referencenumerals as those used for the first embodiment. The same applies to theother embodiments to be described below.

The first embodiment computes coordinates for pattern B from pattern A.As illustrated in FIG. 8, the second embodiment computes coordinates forpattern A from pattern B.

The shapes and placement of the reference camera (left camera 7), theadjustment-targeted camera (front camera 5), and the calibration sheet31 are similar to those of the first embodiment.

FIG. 9 illustrates a calibration method according to the secondembodiment. At S300, the ECU 3 acquires information about the referencecamera and the adjustment-targeted camera based on a signal from thediagnostic tool 27.

At S310, the ECU 3 performs the capture using the front camera 5 servingas the adjustment-targeted camera.

At S320, the ECU 3 detects pattern A similarly to S120 in the firstembodiment.

At S330, similarly to S130 in the first embodiment, the ECU 3 retrieves(computes) and corrects parameters for the front camera 5 based onpattern A. Specifically, the ECU 3 retrieves (computes) and correctsRoll, Pitch, and Z out of parameters X, Y, Z, Roll, Pitch, and Yaw sothat pattern A (in detail, its calibration mark 35) conforms to a 500-mmsquare in terms of the shape and the size.

At S340, the ECU 3 captures pattern B using the left camera 7 serving asthe reference camera that need not adjust the transformation rule.

At S350, the ECU 3 detects pattern B from the captured image.Specifically, because pattern B is square, the ECU 3 detects four apexesCT of the square for the calibration mark 33.

The dimension of the calibration sheet 31 is pre-known. At S360, the ECU3 computes coordinates for provisional pattern A′ (in detail, itscalibration mark) corresponding to coordinates for four apexes CT ofpattern A (in detail, its calibration mark 35) from coordinates for fourapexes CT of pattern B (in detail, its calibration mark 33).

At S370, the ECU 3 approximates provisional pattern A′ to pattern A(specifically, approximates the calibration marks to each other). TheECU 3 retrieves (computes) and corrects parameters for the front camera5 so as to minimize a positional error. Specifically, the ECU 3approximates provisional pattern A′ to pattern A by adjusting theorientation and the position of provisional pattern A. The ECU 3retrieves (computes) and corrects Yaw, X, and Y out of parameters X, Y,Z, Roll, Pitch, and Yaw so as to minimize a positional error.

At S380, parameters X, Y, Z, Roll, Pitch, and Yaw after the retrieval(computation) for the front camera 5 are stored in the memory 23 by theECU 3. Then the processing is ended.

The second embodiment provides the same advantages as the firstembodiment.

(Third Embodiment)

The description below explains a third embodiment, in which anexplanation on contents similar to those in the first embodiment isomitted or simplified.

In the third embodiment, as illustrated in FIG. 10, the capture range ofan adjustment-targeted camera compliant with the transformation rule isnot adjacent to but is distant from the capture range of a referencecamera, for which the adjustment is not needed. The description belowexplains a case in which the calibration sheet 31 is placed so as toconnect the capture ranges that do not overlap with each other. In thisexample, the adjustment-targeted camera is the front camera 5. Thereference camera is the rear camera 11.

The description below explains a calibration method according to thethird embodiment in order.

As illustrated in FIG. 10, the calibration sheet 31 is placed on theground. Specifically, pattern A of the calibration sheet 31 is placed incapture range SF for the front camera 5. Pattern B is placed in capturerange SB for the rear camera 11.

Similarly to the first embodiment, the ECU 3 causes the front camera 5to capture pattern A of the calibration sheet 31.

The ECU 3 detects pattern A from the captured image similarly to thefirst embodiment. Namely, the ECU 3 detects apexes CT of the calibrationmark 35 in pattern A.

Similarly to the first embodiment, the ECU 3 retrieves (computes) andcorrects parameters for the front camera 5 based on pattern A.Specifically, the ECU 3 retrieves (computes) and corrects Roll, Pitch,and Z out of parameters X, Y, Z, Roll, Pitch, and Yaw so that pattern A(in detail, its calibration mark 35) conforms to a 500-mm square interms of the shape and the size.

Similarly to the first embodiment, the ECU 3 computes provisionalpattern B′ (in detail, its calibration mark) corresponding to pattern B(in detail, its calibration mark 33) from four apexes CT of pattern A(in detail, its calibration mark 35) using the dimensions of thecalibration sheet 31.

Similarly to the first embodiment, the ECU 3 causes the rear camera 11serving as the reference camera to capture pattern B.

Similarly to the first embodiment, the ECU 3 detects pattern B from thecaptured image. Specifically, because pattern B (and the calibrationmark 33) is square, the ECU 3 detects four apexes CT of the square forthe calibration mark 35.

Similarly to the first embodiment, the ECU 3 approximates provisionalpattern B′ to pattern B (specifically, approximates the correspondingcalibration marks to each other). The ECU 3 retrieves (computes) andcorrects parameters for the front camera 5 so as to minimize apositional error.

Specifically, the ECU 3 approximates provisional pattern B′ to pattern B(in detail, its calibration mark) by adjusting the orientation and theposition of provisional pattern B′ (in detail, its calibration mark).The ECU 3 retrieves (computes) and corrects Yaw, X, and Y out ofparameters X, Y, Z, Roll, Pitch, and Yaw so as to minimize a positionalerror.

Parameters X, Y, Z, Roll, Pitch, and Yaw after the retrieval(computation) for the front camera 5 are stored by the ECU 3.

The third embodiment provides the same advantages as the firstembodiment.

In particular, the third embodiment provides an advantage of being ableto adjust the front camera 5 or the rear camera 11 even if the sidecamera 7 or 9 is unavailable. Easy adjustment is available even if thecapture ranges do not overlap and separate from each other.

(Fourth Embodiment)

The description below explains a fourth embodiment, in which anexplanation on contents similar to those in the first embodiment isomitted or simplified.

In the fourth embodiment, as illustrated in FIG. 11, the capture rangeof an adjustment-targeted camera compliant with the transformation rulepartially overlaps with the capture range of a reference camerarequiring no adjustment. In this example, the adjustment-targeted camerais the left camera 7. The reference camera is the rear camera 11.

Particularly, the left camera 7, when positioned as illustrated in FIG.11, captures an image of pattern B to be displayed very small asillustrated in FIGS. 12A and 12D that represent images after thedistortion correction. As illustrated in FIGS. 12B and 12E, pattern B isdefocused even if the distortion correction is performed. Apexes CTcannot be detected accurately. However, the fourth embodiment accuratelydetects apexes CT using the method described below.

As will be described later, the fourth embodiment enables pattern A tobe sufficiently large as illustrated in FIGS. 12C and 12F. Becausedisplayed images are clear and the dimensions of the calibration sheet31 are pre-known, the position of pattern B can be computed from theposition of pattern A.

The description below explains a calibration method according to thefourth embodiment in order.

As illustrated in FIG. 11, the calibration sheet 31 is placed on theground. Specifically, pattern A of the calibration sheet 31 is placed incapture range SL for the left camera 7. Pattern B is placed in capturerange SB for the rear camera 11. Pattern B is placed not to enteroverlapping range SLB between capture range SL for the left camera 7 andcapture range SB for the rear camera 11. Pattern B is placed to enteroverlapping range SLB.

Similarly to the first embodiment (but using the different camera), theECU 3 causes the left camera 7 to capture pattern A of the calibrationsheet 31.

The ECU 3 detects pattern A from the captured image similarly to thefirst embodiment. Namely, the ECU 3 detects apexes CT of the calibrationmark 35 in pattern A.

Similarly to the first embodiment, the ECU 3 retrieves (computes) andcorrects parameters for the left camera 7. Specifically, the ECU 3retrieves (computes) and corrects Roll, Pitch, and Z out of parametersX, Y, Z, Roll, Pitch, and Yaw so that pattern A (in detail, itscalibration mark 35) conforms to a 500-mm square in terms of the shapeand the size.

Similarly to the first embodiment, the ECU 3 computes provisionalpattern B′ (in detail, its calibration mark) corresponding to pattern B(in detail, its calibration mark 33) from four apexes CT of pattern A(in detail, its calibration mark 35) using the dimensions of thecalibration sheet 31.

Similarly to the first embodiment, the ECU 3 causes the rear camera 11serving as the reference camera to capture pattern B.

Similarly to the first embodiment, the ECU 3 detects pattern B from thecaptured image. Specifically, because pattern B (and the calibrationmark 33) is square, the ECU 3 detects four apexes CT of the square forthe calibration mark 35.

Similarly to the first embodiment, the ECU 3 approximates pattern B′ topattern B (specifically, approximates the corresponding calibrationmarks to each other). The ECU 3 retrieves (computes) and correctsparameters for the left camera 7 so as to minimize a positional error.

Specifically, the ECU 3 approximates pattern B′ to pattern B (in detail,its calibration mark 33) by adjusting the orientation and the positionof pattern B′ (in detail, its calibration mark). The ECU 3 retrieves(computes) and corrects Yaw, X, and Y out of parameters X, Y, Z, Roll,Pitch, and Yaw so as to minimize a positional error.

Parameters X, Y, Z, Roll, Pitch, and Yaw after the retrieval(computation) for the left camera 7 are stored by the ECU 3.

Similarly to the first embodiment, the fourth embodiment can alsoappropriately calibrate the left camera 7 serving as anadjustment-targeted camera.

In the fourth embodiment, pattern A is advantageously sufficiently largeeven if the left camera 7 inaccurately captures pattern B. Displayedimages are clear. The dimensions of the calibration sheet 31 arepre-known. Therefore, the position of pattern B can be easily computedfrom the position of pattern A.

The fourth embodiment places pattern B in overlapping range SLB betweenthe left camera 7 and the rear camera 11. Similarly to the firstembodiment, however, pattern B may be calibrated by placing pattern B incapture range SB with which the rear camera 11 does not overlap.

(Fifth Embodiment)

The description below explains a fifth embodiment, in which anexplanation on contents similar to those in the first embodiment isomitted or simplified.

This embodiment collectively explains a method of calibrating onethrough three cameras by changing positions of the single calibrationsheet 31 as illustrated in FIG. 13.

a) The description below first explains an example of placing thecalibration sheet 31 when the front camera 5 is used as anadjustment-targeted camera. The other cameras are used as referencecameras.

As illustrated on the left of the front row in FIG. 13, pattern A isplaced in the capture range for the front camera 5 where the capturerange does not overlap with the capture range for the left camera 7.Pattern B is placed in the capture range for the left camera 7 where thecapture range does not overlap with the capture range for the frontcamera 5.

As illustrated on the right of the front row in FIG. 13, pattern A isplaced in the capture range for the front camera 5 where the capturerange does not overlap with the capture range for the right camera 9.Pattern B is place in the capture range for the right camera 9 where thecapture range does not overlap with the capture range for the frontcamera 5.

Similarly to the first embodiment, the ECU 3 performs the calibrationusing data for patterns A and B.

b) The description below explains an example of placing the calibrationsheet 31 when the rear camera 11 is used as an adjustment-targetedcamera. The other cameras are used as reference cameras.

As illustrated on the left of the rear row in FIG. 13, pattern B isplaced in the capture range for the left camera 7 where the capturerange does not overlap with the capture range for the rear camera 7.Pattern A is place in the capture range for the rear camera 11 where thecapture range does not overlap with the capture range for the leftcamera 7.

As illustrated on the right of the rear row in FIG. 13, pattern B isplaced in the capture range for the right camera 9 where the capturerange does not overlap with the capture range for the rear camera 11.Pattern A is place in the capture range for the rear camera 11 where thecapture range does not overlap with the capture range for the rightcamera 9.

Similarly to the first embodiment, the ECU 3 performs the calibrationusing data for patterns A and B.

c) The description below explains an example of placing the calibrationsheet 31 when the left camera 7 is used as an adjustment-targetedcamera. The other cameras are used as reference cameras.

As illustrated on the left of the left row in FIG. 13, pattern B isplaced in the capture range for the front camera 5 where the capturerange does not overlap with the capture range for the left camera 7.Pattern A is place in the capture range for the left camera 7 where thecapture range does not overlap with the capture range for the frontcamera 5.

As illustrated on the right of the left row in FIG. 13, pattern A isplaced in the capture range for the left camera 7 where the capturerange does not overlap with the capture range for the rear camera 11.Pattern B is place in the capture range for the rear camera 11 where thecapture range does not overlap with the capture range for the leftcamera 7.

Similarly to the first embodiment, the ECU 3 performs the calibrationusing data for patterns A and B.

d) The description below explains an example of placing the calibrationsheet 31 when the right camera 9 is used as an adjustment-targetedcamera. The other cameras are used as reference cameras.

As illustrated on the left of the right row in FIG. 13, pattern B isplaced in the capture range for the front camera 5 where the capturerange does not overlap with the capture range for the right camera 9.Pattern A is place in the capture range for the right camera 9 where thecapture range does not overlap with the capture range for the frontcamera 5.

As illustrated on the right of the right row in FIG. 13, pattern A isplaced in the capture range for the right camera 9 where the capturerange does not overlap with the capture range for the rear camera 11.Pattern B is place in the capture range for the rear camera 11 where thecapture range does not overlap with the capture range for the rightcamera 9.

Similarly to the first embodiment, the ECU 3 performs the calibrationusing data for patterns A and B.

e) When two adjustment-targeted cameras are available, the ECU 3 cancalibrate each adjustment-targeted camera using the reference cameraadjacent to each adjustment-targeted camera similarly to the firstembodiment.

f) When three adjustment-targeted cameras are available, the ECU 3 cansequentially calibrate each adjustment-targeted camera using onereference camera similarly to the first embodiment.

For example, suppose that the front camera 5 is used as the referencecamera and the left camera 7, the right camera 9, and the rear camera 11are used as adjustment-targeted cameras. In such a case, for example,the ECU 3 calibrates the left camera 7 using the front camera 5. Thischanges the left camera 7 to the reference camera.

The ECU 3 calibrates the rear camera 11 using the left camera 7 servingas the reference camera.

The ECU 3 calibrates the right camera 9 using the rear camera 11 servingas the reference camera.

This can calibrate three adjustment-targeted cameras.

(Sixth Embodiment)

The description below explains a sixth embodiment, in which anexplanation on contents similar to those in the first embodiment isomitted or simplified.

The sixth embodiment describes a calibration method used when all thecameras are adjustment-targeted cameras that require adjusting thetransformation rule for projection transform. This may occur when, forexample, all the four cameras are replaced or the ECU 3 is replaced.

a) The description below explains a method of placing the calibrationsheet 31.

As illustrated in FIG. 14, the sixth embodiment places the fourcalibration sheets 31 (first through fourth calibration sheets 31 athrough 31 d).

The four calibration sheets 31 are placed, so that defocusing due tocalibration patterns A through H distanced too far from optical axes ofthe cameras 5 through 11 does not occur.

Firstly (1st time), pattern A of the first calibration sheet 31 a isplaced only in capture range SF for the front camera 5 outside anoverlap between capture ranges SF and SL for the front camera 5 and theleft camera 7 (i.e., outside the capture ranges for the other cameras).Pattern B of the first calibration sheet 31 a is placed only in capturerange SL for the left camera 7 outside an overlap between capture rangesSF and SL for the front camera 5 and the left camera 7.

Secondly (2nd time), pattern C (similar to pattern A) of the secondcalibration sheet 31 b is placed only in capture range SL for the leftcamera 7 outside an overlap between capture ranges SL and SB for theleft camera 7 and the rear camera 11. Pattern D (similar to pattern B)of the second calibration sheet 31 b is placed only in capture range SBfor the rear camera 11 outside an overlap between capture ranges SL andSB for the left camera 7 and the rear camera 11.

Thirdly (3rd time), pattern E (similar to pattern A) of the thirdcalibration sheet 31 c is placed only in capture range SB for the rearcamera 11 outside an overlap between capture ranges SB and SR for therear camera 11 and the right camera 9. Pattern F (similar to pattern B)of the third calibration sheet 31 c is placed only in capture range SRfor the right camera 9 outside an overlap between capture ranges SB andSR for the rear camera 11 and the right camera 9.

Lastly (4th time), pattern G (similar to pattern A) of the fourthcalibration sheet 31 d is placed only in capture range SR for the rightcamera 9 outside an overlap between capture ranges SR and SF for theright camera 9 and the front camera 5. Pattern H (similar to pattern B)of the fourth calibration sheet 31 d is placed only in capture range SFfor the front camera 5 outside an overlap between capture ranges SR andSF for the right camera 9 and the front camera 5.

The sixth embodiment places the calibration sheets 31 in theabove-mentioned order for illustrative purposes. However, the order maybe changed.

b) The description below explains a calibration method according to thesixth embodiment.

The description below explains a case of capturing images in the orderof the front camera 5, the left camera 7, the rear camera 11, and theright camera 9 though the order is not limited to this,

At S400, as illustrated in FIGS. 15A and 15B, the ECU 3 receives aninstruction to calibrate all the four cameras 5 through 11 based on asignal from the diagnostic tool 27.

At S410, the ECU 3 causes the front camera 5 to capture pattern A forthe first calibration sheet 31 a and pattern H for the fourthcalibration sheet 31 d.

At S420, the ECU 3 detects patterns A and H from the captured imagessimilarly to the first embodiment.

At S430, the ECU 3 retrieves (computes) and corrects parameters for thefront camera 5 based on patterns A and H similarly to the firstembodiment.

Specifically, the ECU 3 retrieves (computes) and corrects Roll, Pitch,and Z out of parameters X, Y, Z, Roll, Pitch, and Yaw so that patterns Aand H (in detail, their calibration patterns) each conform to a 500-mmsquare in terms of the shape and the size.

The dimensions of the first calibration sheet 31 a and the fourthcalibration sheet 31 d are pre-known. At S440, the ECU 3 computescoordinates corresponding to four apexes of patterns B and G (in detail,their calibration patterns) as squares from coordinates for patterns Aand H (in detail, their calibration patterns) as squares. Namely, theECU 3 computes the coordinates as provisional pattern B′ and provisionalpattern G′ (in detail, their calibration patterns).

At S450, the ECU 3 causes the left camera 7 to capture patterns B and C.

At S460, similarly to S420, the ECU 3 detects patterns B and C.

At S470, similarly to S430, the ECU 3 retrieves (computes) and correctsparameters for the left camera 7 based on patterns B and C.

Specifically, the ECU 3 retrieves (computes) and corrects Roll, Pitch,and Z out of parameters X, Y, Z, Roll, Pitch, and Yaw so that patterns Band C (in detail, their calibration patterns) each conform to a 500-mmsquare in terms of the shape and the size.

The dimensions of the first calibration sheet 31 a and the secondcalibration sheet 31 b are pre-known. At S480, the ECU 3 computescoordinates corresponding to four apexes of patterns A and D (in detail,their calibration patterns) as squares from coordinates for four apexesof patterns B and C (in detail, their calibration patterns) as squares.Namely, the ECU 3 computes the coordinates as provisional pattern A′ andprovisional pattern D′ (in detail, their calibration patterns).

At S490, the ECU 3 allows the rear camera 11 to capture patterns D andE.

At S500, similarly to S420, the ECU 3 detects patterns D and E.

At S510, the ECU 3 retrieves (computes) and corrects parameters for therear camera 11 based on patterns D and E.

Specifically, the ECU 3 retrieves (computes) and corrects Roll, Pitch,and Z out of parameters X, Y, Z, Roll, Pitch, and Yaw so that patterns Dand E (in detail, their calibration patterns) each conform to a 500-mmsquare in terms of the shape and the size.

The dimensions of the second calibration sheet 31 b and the thirdcalibration sheet 31 c are pre-known. At S520, the ECU 3 computescoordinates corresponding to four apexes of patterns C and F (in detail,their calibration patterns) as squares from coordinates for four apexesof patterns D and E (in detail, their calibration patterns) as squares.Namely, the ECU 3 computes the coordinates as provisional pattern C′ andprovisional pattern F′ (in detail, their calibration patterns).

At S530, the ECU 3 causes the right camera 9 to capture patterns F andG.

At S540, similarly to S420, the ECU 3 detects patterns F and G.

At S550, the ECU 3 retrieves (computes) and corrects parameters for theright camera 9 based on patterns F and G.

Specifically, the ECU 3 retrieves (computes) and corrects Roll, Pitch,and Z out of parameters X, Y, Z, Roll, Pitch, and Yaw so that patterns Fand G (in detail, their calibration patterns) each conform to a 500-mmsquare in terms of the shape and the size.

The dimensions of the third calibration sheet 31 c and the fourthcalibration sheet 31 d are pre-known. At S560, the ECU 3 computescoordinates corresponding to four apexes of patterns E and H (in detail,their calibration patterns) as squares from coordinates for four apexesof patterns F and G (in detail, their calibration patterns) as squares.Namely, the ECU 3 computes the coordinates as provisional pattern E′ andprovisional pattern H′ (in detail, their calibration patterns).

The sixth embodiment computes coordinates for four apexes of a square inunits of two patterns. However, the coordinates may be found for eachpattern one by one.

At S570, the ECU 3 approximates pattern A′, pattern B′, pattern C′,pattern D′, pattern E′, pattern F′, pattern G′, and pattern H′ (indetail, their calibration patterns) to pattern A, pattern B, pattern C,pattern D, pattern E, pattern F, pattern G, and pattern H (in detail,their calibration patterns), respectively. The ECU 3 retrieves(computes) and corrects parameters for the front camera 5, the leftcamera 7, the right camera 9, and the rear camera 11 so as to minimize apositional error.

Specifically, similarly to the first embodiment, the ECU 3 adjustsorientations and positions of pattern A′, pattern B′, pattern C′,pattern D′, pattern E′, pattern F′, pattern G′, pattern H′ (in detail,their calibration patterns) to approximate to pattern A, pattern B,pattern C, pattern D, pattern E, pattern F, pattern G, pattern H (indetail, their calibration patterns), respectively. The ECU 3 retrieves(computes) and corrects Yaw, X, and Y out of parameters X, Y, Z, Roll,Pitch, and Yaw for the cameras 5 through 11 so as to minimize apositional error.

For example, the correction may be performed in the order as describedbelow.

Similarly to the first embodiment (or the second embodiment), theprocess of adjusting an adjacent camera using one calibration sheet 31is sequentially repeated basically. Similarly to the first embodiment,matching coordinates for a provisional adjustment calibration pattern(e.g., pattern B′) with coordinates for an adjustment calibrationpattern (e.g., pattern B) may be performed. Similarly to the secondembodiment, matching coordinates for a provisional reference calibrationpattern (e.g., pattern A′) with coordinates for a provisional referencecalibration pattern (e.g., pattern A) may be performed.

The description of “in detail, its calibration pattern (or theircalibration patterns)” may be omitted from the explanation below abouteach pattern.

Specifically, the ECU 3 first assumes the front camera 5 to be aprovisional reference camera. The ECU 3 approximates pattern A′ computedfrom pattern B to pattern A and minimizes a positional error to retrieve(compute) and correct Yaw, X, and Y for the left camera 7.

The ECU 3 assumes the left camera 7 to be a provisional referencecamera. The ECU 3 approximates pattern C′ computed from pattern D topattern C and minimizes a positional error to retrieve (compute) andcorrect Yaw, X, and Y for the rear camera 11.

The ECU 3 assumes the rear camera 11 to be a provisional referencecamera. The ECU 3 approximates pattern E′ computed from pattern F topattern E and minimizes a positional error to retrieve (compute) andcorrect Yaw, X, and Y for the right camera 9.

Finally, the ECU 3 assumes the right camera 9 to be a provisionalreference camera. The ECU 3 approximates pattern G′ computed frompattern H to pattern G and minimizes a positional error to retrieve(compute) and correct Yaw, X, and Y for the front camera 5.

The ECU 3 retrieves (computes) and corrects Yaw, X, and Y for all thefour cameras 5 through 11 from the left camera 7 to the front camera 5.The ECU 3 follows the correction cycle, adjusts orientations andpositions of patterns A through H, and minimizes a positional error toretrieve (compute) and correct Yaw, X, and Y for the cameras 5 through11.

The order of the correction procedures is merely an example and is notlimited thereto.

Finally, at S580, the ECU 3 causes the memory 23 to store parameters X,Y, Z, Roll, Pitch, Yaw after retrieving (computing) the front camera 5,the left camera 7, the right camera 9, and the rear camera 11 and onceterminates the process.

c) The description below explains a subsequent process (post-process).

The post-process transforms the coordinate system for the four cameras 5through 11 into the vehicle coordinate system.

The process from S400 to S580 adjusts parameters with reference to aprovisional reference camera (such as the front camera 5). Completingthe adjustment process reveals a positional relationship among the fourcameras 5 through 11 with reference to the provisional reference camera.Namely, the process reveals parameters such as coordinates in aprovisional reference camera coordinate system as illustrated in FIG.16. However, the calibration aims at finding parameters in the vehiclecoordinate system, not values with reference to the provisionalreference camera. The provisional reference camera coordinate systemneeds to be transformed into the vehicle coordinate system.

The description below explains the process of transformation into thevehicle coordinate system as the post-process in accordance with steps 1through 3 described below in order.

<Step 1>

As illustrated in FIG. 17A, the ECU 3 finds angle θ1 between the frontcamera 5 and the rear camera 11 in the vehicle coordinate system usingdesign values (x1, y1) and (x2, y2) for camera positions (e.g.,positions of the front camera 5 and the rear camera 11). The broken lineindicates the front-back direction of the vehicle (passing through theposition of the camera 11).

As illustrated in FIG. 17B (provisional reference camera coordinatesystem), the ECU 3 finds θ2 between the front camera 5 and the rearcamera 11 in the provisional reference camera coordinate system usingpositions (x1′, y1′) and (x2′, y2′) of the front camera 5 and the rearcamera 11 in the provisional reference camera coordinate system. Thebroken line corresponds to an optical axis of the camera 5.

The ECU 3 finds rotation angle R using equation (1) below.R=θ1−θ2  (1)

<Step 2>

As illustrated in FIG. 18A (provisional reference camera coordinatesystem), the ECU 3 rotates (x2′, y2′) with reference to the origin inthe provisional reference camera coordinate system using rotation angleR found by equation (1), thereby giving the coordinate after rotation as(x2″, y2″).

FIG. 18B illustrates the vehicle coordinate system and the provisionalreference camera coordinate system overlapped with each other. MidpointM is defined between (x1, y1) and (x2, y2). Midpoint M′ is definedbetween (x1′, y1′) and (x2″, y2″). The ECU 3 finds an X-directiondistance and a Y-direction distance for M and M′, thereby giving thedistances as OFFSET_X and OFFSET_Y, respectively.

<Step 3>

The ECU 3 adds rotation angle R, OFFSET_X, and OFFSET_Y found at steps 1and 2 to corresponding values of Yaw, X, and Y for the cameras 5 through11 with reference to each provisional reference camera. This completesthe transformation into the vehicle coordinate system.

As described above in detail, the sixth embodiment can calibrate all thecameras 5 through 11 even when all the four cameras 5 through 11 arereplaced or the ECU 3 is replaced.

(Seventh Embodiment)

The description below explains a seventh embodiment, in whichexplanation on contents similar to those in the first embodiment isomitted or simplified.

The seventh embodiment performs calibration using the single calibrationsheet containing three patterns A B, and C (along with correspondingcalibration marks).

FIG. 19A illustrates a method of estimating pattern B from pattern A.The method finds two lines L1 and L2 using four apexes CT for thecalibration mark of pattern A. The vehicle travels a predetermineddistance in the corresponding direction to find apexes CT for thecalibration mark of pattern B. The method is similar to the firstembodiment.

The accuracy of lines L1 and L2 depends on the accuracy of pattern A(accordingly, of its apex CT for the calibration mark).

In this regard, as illustrated in FIG. 19B, the seventh embodiment usesa calibration sheet containing patterns A, C, and B provided in thelonger direction. Patterns C and B are shaped similarly to pattern A.

Pattern C is provided as a graphic comparable to pattern A translated topattern B along the longer direction of the calibration sheet 31.Pattern C is positioned to be intermediate between patterns A and B, forexample.

The use of patterns A and C (in detail, apexes CT for the correspondingcalibration marks) can improve the accuracy of lines L1 and L2. As aresult, the accuracy of estimating pattern B can be further improved.

A publicly known least-square method can be used to find lines L1 andL2. The method minimizes a distance between line L1 (or line L2) andapexes CT for the calibration marks of corresponding patterns A and C.

Obviously, the present disclosure is not limited to the above-mentionedembodiments but may be variously embodied.

In each of the above-mentioned embodiments, for example, a functionincluded in one component may be distributed to several components.Functions included in several components may be integrated into onecomponent. At least part of the configuration of the embodiment may bereplaced by a publicly known configuration that includes a comparablefunction. At least part of the configuration of the embodiment may beadded to or may replace the configuration of another embodiment.

The above-mentioned calibration sheet may be available as not only afoldable calibration sheet, but also a plate-like calibration memberthat cannot be folded but can be bent.

Colors of the above-mentioned calibration sheet are not limited tospecific ones, so that the above-mentioned calibration is available.

The end weight may be omitted, so that the above-mentioned calibrationis available.

An onboard periphery image display device in a first example of thepresent disclosure is mounted on an automobile and comprises a pluralityof cameras, a vehicle periphery image generation portion, a displayapparatus, and an adjustment portion. The plurality of cameras aremounted on an automobile to image a periphery of the automobile. Inaccordance with a transformation rule determined for each cameraprovided to capture the image, the vehicle periphery image generationportion applies projection transform to the images captured by thecameras to generate a vehicle periphery image synthesized in a singleimage space. The vehicle periphery image represents the periphery of theautomobile observed from a specified viewpoint. The display apparatusdisplays the vehicle periphery image generated by the vehicle peripheryimage generation portion. The adjustment portion adjusts thetransformation rule determined for each of the cameras. In a situationwhere: a calibration member, in which a plurality of calibrationpatterns with predetermined sizes and shapes have a predeterminedpositional relationship, is placed around the automobile; apredetermined calibration pattern of the calibration patterns is placedin a capture area of a reference camera, which is a camera serving as areference to adjust the transformation rule; and another calibrationpattern of the calibration patterns is placed in a capture area of anadjustment-targeted camera, which is a camera targeted for adjustment ofthe transformation rule, the adjustment portion: detects a coordinate ofan image of a reference calibration pattern through applying theprojection transform to the image of the predetermined calibrationpattern of the calibration member captured by the reference camera inaccordance with the transformation rule determined for the referencecamera; detects an coordinate of an image of an adjustment calibrationpattern through applying the projection transform to the image of theanother calibration pattern of the calibration member captured by theadjustment-targeted camera in accordance with the transformation ruledetermined for the adjustment-targeted camera; performs detecting acoordinate of an image of a provisional reference calibration patterncorresponding to the reference calibration pattern based on a positionalrelationship of the coordinate of the image of the adjustmentcalibration pattern with the calibration, patterns or detecting acoordinate of an image of a provisional adjustment calibration patterncorresponding to the adjustment calibration pattern based on apositional relationship of the coordinate of the image of the referencecalibration pattern with the calibration patterns; and adjusts thetransformation rule determined for the adjustment-targeted camera sothat the coordinate of the image of the provisional referencecalibration pattern matches the coordinate of the image of the referencecalibration pattern or the coordinate of the image of the provisionaladjustment calibration pattern matches the coordinate of the image ofthe adjustment calibration pattern.

A camera adjustment method in a first example of the present disclosureis provided for an onboard periphery image display device including aplurality of cameras that are mounted on an automobile to image aperiphery of the automobile; a vehicle periphery image generationportion that, in accordance with a transformation rule determined foreach camera provided to capture the image, applies projection transformto the images captured by the cameras to generate a vehicle peripheryimage synthesized in a single image space, wherein the vehicle peripheryimage represents the periphery of the automobile observed from aspecified viewpoint; and a display apparatus that displays the vehicleperiphery image generated by the vehicle periphery image generationportion, and the camera adjustment method is provided for adjusting thetransformation rules for the cameras. In a situation where: acalibration member, in which a plurality of calibration patterns withpredetermined sizes and shapes have a predetermined positionalrelationship, is placed around the automobile; a predeterminedcalibration pattern of the calibration patterns is placed in a capturearea of a reference camera, which is a camera serving as a reference toadjust the transformation rule; and another calibration pattern of thecalibration patterns is placed in a capture area of anadjustment-targeted camera, which is a camera targeted for adjustment ofthe transformation rule, the camera adjustment method comprisesdetecting a coordinate of an image of a reference calibration patternthrough applying the projection transform to the image of thepredetermined calibration pattern of the calibration member captured bythe reference camera in accordance with the transformation ruledetermined for the reference camera; detecting an coordinate of an imageof an adjustment calibration pattern through applying the projectiontransform to the image of the another calibration pattern of thecalibration member captured by the adjustment-targeted camera inaccordance with the transformation rule determined for theadjustment-targeted camera; performing detecting a coordinate of animage of a provisional reference calibration pattern corresponding tothe reference calibration pattern based on a positional relationship ofthe coordinate of the image of the adjustment calibration pattern withthe calibration patterns or detecting a coordinate of an image of aprovisional adjustment calibration pattern corresponding to theadjustment calibration pattern based on a positional relationship of thecoordinate of the image of the reference calibration pattern with thecalibration patterns; and adjusting the transformation rule determinedfor the adjustment-targeted camera so that the coordinate of the imageof the provisional reference calibration pattern matches the coordinateof the image of the reference calibration pattern or the coordinate ofthe image of the provisional adjustment calibration pattern matches thecoordinate of the image of the adjustment calibration pattern.

As illustrated in FIG. 3, for example, the calibration member containingtwo or more calibration patterns at the predetermined positions may beplaced so as to position the calibration patterns in the correspondingcapture ranges for the reference camera and the adjustment-targetedcamera. After that, coordinates for the reference calibration pattern(B) may be detected from an image captured by the reference camera. Animage of the adjustment calibration pattern (A) may be detected from animage captured by the adjustment-targeted camera.

There may be a case of detecting coordinates for the image of theprovisional reference calibration pattern (B′) corresponding to thereference calibration pattern (B) based on the positional relationshipamong coordinates for the image of the adjustment calibration pattern(A) and both calibration patterns (A and B). In such a case, thetransformation rule predetermined for the adjustment-targeted camera maybe adjusted so that coordinates for the image of the provisionalreference calibration pattern (B′) match coordinates for the image ofthe reference calibration pattern (B).

Alternatively, there may be a case of detecting coordinates for theimage of the provisional adjustment calibration pattern (A′)corresponding to the adjustment calibration pattern (A) based on thepositional relationship among coordinates for the image of the referencecalibration pattern (B) and both calibration patterns (A and B). In sucha case, the transformation rule predetermined for theadjustment-targeted camera may be adjusted so that coordinates for theimage of the provisional adjustment calibration pattern (A′) matchcoordinates for the image of the adjustment calibration pattern (A).

Even if the cameras' capture ranges do not overlap, the onboardperiphery image display device and the camera adjustment methodaccording to the present disclosure can easily calibrateadjustment-targeted cameras by placing the calibration patterns in thecapture ranges and performing the above-mentioned process.

In a conventional one, a screen may display a single pattern to be verysmall. In such a case, the onboard periphery image display device andthe camera adjustment method according to the present disclosure usesthe calibration member containing different calibration patterns placedin capture ranges for the different cameras to improve the accuracy forthe cameras to detect calibration patterns. The onboard periphery imagedisplay device and the camera adjustment method can provide an effect ofaccurately performing the calibration by performing the above-mentionedprocess.

An onboard periphery image display device in a second example of thepresent disclosure is mounted on an automobile and comprises a pluralityof cameras, a vehicle periphery image generation portion, a displayapparatus and an adjustment portion. The plurality of cameras aremounted on an automobile to image a periphery of the automobile. Inaccordance with a transformation rule determined for each cameraprovided to capture the image, the vehicle periphery image generationportion applies projection transform to the images captured by thecameras to generate a vehicle periphery image synthesized in a singleimage space. The vehicle periphery image represents the periphery of theautomobile observed from a specified viewpoint. The display apparatusdisplays the vehicle periphery image generated by the vehicle peripheryimage generation portion. The adjustment portion adjusts thetransformation rule determined for each of the cameras. In a situationwhere: calibration members, in which a plurality of calibration patternswith predetermined sizes and shapes have a predetermined positionalrelationship, are placed around the automobile; a predeterminedcalibration pattern of the calibration patterns is placed in a capturearea of a predetermined camera; and another calibration pattern of thecalibration patterns is placed in a capture area of another camera, theadjustment portion: of one set of two adjustment-targeted cameras thatare capable of independently capturing two calibration patterns of apredetermined calibration member of the calibration members, uses onecamera as a provisional reference camera and the other camera as anadjustment-targeted camera; detects an coordinate of an image of aprovisional reference calibration pattern (B) through, in accordancewith the transformation rule determined for the provisional referencecamera, applying the projection transform to the image of apredetermined calibration pattern of the two calibration patternscaptured by the provisional reference camera; detects a coordinate of animage of an adjustment calibration pattern through, in accordance withthe transformation rule determined for the adjustment-targeted camera,applying the projection transform to the image of the other calibrationpattern of the two calibration patterns captured by theadjustment-targeted camera; performs detecting a coordinate of an imageof a provisional reference calibration pattern (B′) corresponding to theprovisional reference calibration pattern (B) based on a positionalrelationship of the coordinate of the image of the adjustmentcalibration pattern with the two calibration patterns or detecting acoordinate of an image of a provisional adjustment calibration patterncorresponding to the adjustment calibration pattern based on apositional relationship of the coordinate of the image of theprovisional reference calibration pattern with the two calibrationpatterns; and adjusts the transformation rule determined for theadjustment-targeted camera so that the coordinate of the image of theprovisional reference calibration pattern (B) matches the coordinate ofthe image of the provisional reference calibration pattern (B′) or thecoordinate of the image of the provisional adjustment calibrationpattern matches the coordinate of the image of the adjustmentcalibration pattern. Said set-by-set-basis adjustment is applied toadjust the transformation rules for all the cameras.

A camera adjustment method in a second example of the present disclosureis provided for an onboard periphery image display device including: aplurality of cameras that are mounted on an automobile to image aperiphery of the automobile; a vehicle periphery image generationportion that, in accordance with a transformation rule determined foreach camera provided to capture the image, applies projection transformto the images captured by the cameras to generate a vehicle peripheryimage synthesized in a single image space, wherein the vehicle peripheryimage represents the periphery of the automobile observed from aspecified viewpoint; and a display apparatus that displays the vehicleperiphery image generated by the vehicle periphery image generationportion. The camera adjustment method is provided for adjusting thetransformation rules for the cameras. In a situation where: calibrationmembers, in which a plurality of calibration patterns with predeterminedsizes and shapes have a predetermined positional relationship, areplaced around the automobile; a predetermined calibration pattern of thecalibration patterns is placed in a capture area of a predeterminedcamera; and another calibration pattern of the calibration patterns isplaced in a capture area of another camera, the camera adjustment methodcomprises: of one set of two adjustment-targeted cameras that arecapable of independently capturing two calibration patterns of apredetermined calibration member of the calibration members, using onecamera as a provisional reference camera and the other camera as anadjustment-targeted camera; detecting an coordinate of an image of aprovisional reference calibration pattern (B) through, in accordancewith the transformation rule determined for the provisional referencecamera, applying the projection transform to the image of apredetermined calibration pattern of the two calibration patternscaptured by the provisional reference camera; detecting a coordinate ofan image of an adjustment calibration pattern through, in accordancewith the transformation rule determined for the adjustment-targetedcamera, applying the projection transform to the image of the othercalibration pattern of the two calibration patterns captured by theadjustment-targeted camera; performing detecting a coordinate of animage of a provisional reference calibration pattern (B′) correspondingto the provisional reference calibration pattern (B) based on apositional relationship of the coordinate of the image of the adjustmentcalibration pattern with the two calibration patterns or detecting acoordinate of an image of a provisional adjustment calibration patterncorresponding to the adjustment calibration pattern based on apositional relationship of the coordinate of the image of theprovisional reference calibration pattern with the two calibrationpatterns; and adjusting the transformation rule determined for theadjustment-targeted camera so that the coordinate of the image of theprovisional reference calibration pattern (B′) matches the coordinate ofthe image of the provisional reference calibration pattern (B) or thecoordinate of the image of the provisional adjustment calibrationpattern matches the coordinate of the image of the adjustmentcalibration pattern. Said set-by-set-basis adjustment is applied toadjust the transformation rules for all the cameras.

As illustrated in FIG. 14, for example, each calibration member containsseveral calibration patterns (A through H) that are each sized andshaped as specified and are arranged based on specified positionalrelationship. The calibration members are placed around the automobile.The specified calibration patterns (A through H) are placed in captureranges for the specified cameras. The other calibration patterns (Athrough H) are placed in capture ranges for the other cameras. Twoadjustment-targeted cameras may be defined as one set under thecondition that the two adjustment-targeted cameras are capable ofindependently capturing two calibration patterns of the calibrationmember. One of the adjustment-targeted cameras may be assumed to be aprovisional reference camera. The other camera may be assumed to be anadjustment-targeted camera. The transformation rule may be adjusted forall the cameras on a set basis according to the procedure describedbelow.

Coordinates for an image of the provisional reference calibrationpattern (B) may be detected from an image captured by the provisionalreference camera. Coordinates for an image of the adjustment calibrationpattern (A) may be detected from an image captured by theadjustment-targeted camera.

There may be a case of detecting coordinates for an image of theprovisional reference calibration pattern (B′) based on positionalrelationship among coordinates for the image of the adjustmentcalibration pattern (A) and both calibration patterns (A and B). In sucha case, a transformation rule predetermined for the adjustment-targetedcamera may be adjusted so that coordinates for the image of theprovisional adjustment calibration pattern (A′) match coordinates forthe image of the adjustment calibration pattern (A).

There may be a case of detecting coordinates for an image of theprovisional adjustment calibration pattern (A′) based on positionalrelationship among coordinates for the image of the provisionalreference calibration pattern (B) and both calibration patterns (A andB). In such a case, the transformation rule predetermined for theadjustment-targeted camera may be adjusted so that coordinates for theimage of the provisional adjustment calibration pattern (A′) matchcoordinates for the image of the adjustment calibration pattern (A).

There may be a case of adjusting all cameras due to replacement of theelectronic control unit, for example. The onboard periphery imagedisplay device and the camera adjustment method according to the presentdisclosure can easily calibrate adjustment-targeted cameras by placingthe calibration patterns in the corresponding capture ranges andperforming the above-mentioned process even if the cameras' captureranges do not overlap.

In a conventional one, a screen may display a single pattern to be verysmall. In such a case, the onboard periphery image display device andthe camera adjustment method according to the present disclosure usesthe calibration member containing different calibration patterns placedin capture ranges for the different cameras to improve the accuracy forthe cameras to detect calibration patterns. The onboard periphery imagedisplay device and the camera adjustment method can provide an effect ofaccurately performing the calibration by performing the above-mentionedprocess.

The invention claimed is:
 1. An onboard periphery image display devicemounted on an automobile, comprising: a plurality of cameras, includinga reference camera having a first capture area and anadjustment-targeted camera having a second capture area, that aremounted on an automobile to image a periphery of the automobile; avehicle periphery image generator that uses a processor to applyprojection transform to images captured by the cameras in accordancewith transformation rules determined for each camera to generate avehicle periphery image synthesized in a single image space, wherein thevehicle periphery image represents the periphery of the automobileobserved from a specified viewpoint; a display apparatus that displaysthe vehicle periphery image generated by the vehicle periphery imagegenerator; and an adjuster configured to adjust, using the processor,the transformation rule determined for at least the adjustment-targetedcamera, wherein, an elongated calibration member is placed around theautomobile, the elongated calibration member comprising first and secondcalibration patterns with predetermined sizes and shapes separated fromeach other based on a predetermined positional relationship, andplacement of the first calibration pattern is in a region of the firstcapture area that does not overlap with the second capture area andplacement of the second calibration pattern is in a region of the secondcapture area that is outside of the region of the first capture areabeing occupied by the first calibration pattern, and the adjuster isconfigured to: detect a coordinate of an image of a referencecalibration pattern by applying projection transform to an image of thefirst calibration pattern captured by the reference camera in accordancewith the transformation rule determined for the reference camera; detecta coordinate of an image of an adjustment calibration pattern byapplying projection transform to an image of the second calibrationpattern captured by the adjustment-targeted camera in accordance withthe transformation rule determined for the adjustment-targeted camera;detect: a coordinate of an image of a provisional reference calibrationpattern based at least in part on the coordinate of the image of theadjustment calibration pattern and the predetermined positionalrelationship between the first and the second calibration patterns,wherein the coordinate of the image of the provisional referencecalibration pattern corresponds to the reference calibration pattern, ora coordinate of an image of a provisional adjustment calibration patternbased at least in part on the coordinate of the image of the referencecalibration pattern and the predetermined positional relationshipbetween the first and the second calibration patterns, wherein thecoordinate of the image of the provisional adjustment calibrationpattern corresponds to the adjustment calibration pattern; and determinean adjusted transformation rule for the adjustment-targeted camera bymatching the coordinate of the image of the provisional referencecalibration pattern with the coordinate of the image of the referencecalibration pattern, or by matching the coordinate of the image of theprovisional adjustment calibration pattern with the coordinate of theimage of the adjustment calibration pattern; and apply projectiontransform to a subsequent image captured by the adjustment-targetedcamera using the adjusted transformation rule.
 2. The onboard peripheryimage display device according to claim 1, wherein the calibrationmember is a sheet.
 3. The onboard periphery image display deviceaccording to claim 1, wherein the calibration patterns are formed atopposite ends of the elongated calibration member in a longitudinaldirection.
 4. The onboard periphery image display device according toclaim 1, wherein the calibration member is rectangular in plan view. 5.The onboard periphery image display device according to claim 1, whereinthe calibration member is made of fiber having a predetermined or lesscontraction/expansion with temperature or humidity.
 6. The onboardperiphery image display device according to claim 1, wherein thecalibration member is foldable into a scroll.
 7. The onboard peripheryimage display device according to claim 1, wherein rod-shaped weightsare attached at opposite ends of the elongated calibration member in alongitudinal direction.
 8. The onboard periphery image display deviceaccording to claim 7, wherein a material of the weight is resin ormetal.
 9. An onboard periphery image display device mounted on anautomobile, comprising: a plurality of cameras, including a first camerahaving a first capture area and a second camera having a second capturearea, that are mounted on an automobile to image a periphery of theautomobile; a vehicle periphery image generator that uses a processor toapply projection transform to images captured by the cameras inaccordance with transformation rules determined for each camera togenerate a vehicle periphery image synthesized in a single image space,wherein the vehicle periphery image represents the periphery of theautomobile observed from a specified viewpoint; a display apparatus thatdisplays the vehicle periphery image generated by the vehicle peripheryimage generator; and an adjuster that adjusts, using the processor, thetransformation rule determined for each of the cameras, wherein,elongated calibration members are placed around the automobile, theelongated calibration members each comprising a adjustment calibrationpattern (A) and a provisional reference calibration pattern (B) withpredetermined sizes and shapes separated from each other based on apredetermined positional relationship, and placement of the provisionalreference calibration pattern (B) is in a region of the first capturearea that does not overlap with the second capture area, and placementof the adjustment calibration pattern (A) is in a region of the secondcapture area that is outside of the region of the first capture areabeing occupied by the provisional reference calibration pattern (B), andthe adjuster is configured to: detect a coordinate of an image of theprovisional reference calibration pattern (B) by applying projectiontransform to an image of the provisional reference calibration pattern(B) captured by the first camera in accordance with the transformationrule determined for the first camera; detect a coordinate of an image ofthe adjustment calibration pattern (A) by applying projection transformto an image of the adjustment calibration pattern (A) captured by thesecond camera in accordance with the transformation rule determined forthe second camera; detect: a coordinate of an image of a provisionalreference calibration pattern (B′) based at least in part on thecoordinate of the image of the adjustment calibration pattern (A) andthe predetermined positional relationship between the provisionalreference calibration pattern (B) and adjustment calibration pattern(A), wherein the coordinate of the image of the provisional referencecalibration pattern (B′) corresponds to the provisional referencecalibration pattern (B), or a coordinate of an image of a provisionaladjustment calibration pattern (A′) based at least in part on thecoordinate of the image of the provisional reference calibration pattern(B) and the predetermined positional relationship between theprovisional reference calibration pattern (B) and adjustment calibrationpattern (A), wherein the coordinate of the image of the provisionaladjustment calibration pattern (A′) corresponds to the adjustmentcalibration pattern (A); determine an adjusted transformation rule forthe second camera by matching the coordinate of the image of theprovisional reference calibration pattern (B′) with the coordinate ofthe image of the provisional reference calibration pattern (B), or bymatching the coordinate of the image of the provisional adjustmentcalibration pattern (A′) with the coordinate of the image of theadjustment calibration pattern (A); and apply projection transform to asubsequent image captured by the second camera using the adjustedtransformation rule.
 10. A camera adjustment method for adjusting thetransformation rules for at least one of a reference camera having afirst capture area and an adjustment-targeted camera having a secondcapture area, the cameras configured to be mounted on an automobile toimage a periphery of the automobile, the automobile including an onboardperiphery image display device, the display device including: (i) avehicle periphery image generator that uses a processor to applyprojection transform to images captured by the cameras in accordancewith transformation rules determined for each camera to generate avehicle periphery image synthesized in a single image space, wherein thevehicle periphery image represents the periphery of the automobileobserved from a specified viewpoint; and (ii) a display apparatus thatdisplays the vehicle periphery image generated by the vehicle peripheryimage generator, wherein the camera adjustment method comprises: placingan elongated calibration member around the automobile, the elongatedcalibration member comprising first and second calibration patterns withpredetermined sizes and shapes separated from each other based on apredetermined positional relationship, and placement of the firstcalibration pattern is in a region of the first capture area that doesnot overlap with the second capture area, and placement of the secondcalibration pattern is in a region of the second capture area that isoutside of the region of the first capture area being occupied by thefirst calibration pattern; detecting a coordinate of an image of areference calibration pattern by applying projection transform to animage of the first calibration pattern captured by the reference camerain accordance with the transformation rule determined for the referencecamera; detecting a coordinate of an image of an adjustment calibrationpattern by applying projection transform to an image of the secondcalibration pattern captured by the adjustment-targeted camera inaccordance with the transformation rule determined for theadjustment-targeted camera; detecting: a coordinate of an image of aprovisional reference calibration pattern based at least in part on theimage of the adjustment calibration pattern and the predeterminedpositional relationship between the first and the second calibrationpatterns, wherein the coordinate of the image of the provisionalreference calibration pattern corresponds to the reference calibrationpattern, or a coordinate of an image of a provisional adjustmentcalibration pattern based at least in part on the coordinate of theimage of the reference calibration pattern and the predeterminedpositional relationship between the first and the second calibrationpatterns, wherein the coordinate of the image of the provisionalreference calibration pattern corresponds to the reference calibrationpattern; determining an adjusted transformation rule for theadjustment-targeted camera by matching the coordinate of the image ofthe provisional reference calibration pattern with the coordinate of theimage of the reference calibration pattern, or by matching thecoordinate of the image of the provisional adjustment calibrationpattern with the coordinate of the image of the adjustment calibrationpattern; and applying projection transform to a subsequent imagecaptured by the adjustment-targeted camera using the adjustedtransformation rule.
 11. A camera adjustment method for adjusting thetransformation rules for at least one camera of a first camera having afirst capture area and a second camera having a second capture area, thecameras configured to be mounted on an automobile to image a peripheryof the automobile, the automobile including an onboard periphery imagedisplay device, the display device including: (i) a vehicle peripheryimage generator that uses a processor to apply projection transform toimages captured by the cameras in accordance with transformation rulesdetermined for each camera to generate a vehicle periphery imagesynthesized in a single image space, wherein the vehicle periphery imagerepresents the periphery of the automobile observed from a specifiedviewpoint; and (ii) a display apparatus that displays the vehicleperiphery image generated by the vehicle periphery image generator,wherein the camera adjustment method comprises, placing elongatedcalibration members around the automobile, the elongated calibrationmembers each comprising an adjustment calibration pattern (A) and aprovisional reference calibration pattern (B) with predetermined sizesand shapes separated from each other based on a predetermined positionalrelationship, and placement of the provisional reference calibrationpattern is in a region of the first capture area that does not overlapwith the second capture area, and placement of the adjustmentcalibration pattern is in a region of the second capture area that isoutside of the region of the first capture area being occupied by theprovisional reference calibration pattern; detecting a coordinate of animage of a provisional reference calibration pattern (B) by applyingprojection transform to an image of the provisional referencecalibration pattern (B) captured by the first camera in accordance withthe transformation rule determined for the first camera; detecting acoordinate of an image of an adjustment calibration pattern (A) byapplying projection transform to an image of the adjustment calibrationpattern (A) captured by the second camera in accordance with thetransformation rule determined for the second camera; detecting: acoordinate of an image of a provisional reference calibration pattern(B′) based at least in part on the coordinate of the image of theadjustment calibration pattern (A) and the predetermined positionalrelationship between the adjustment calibration pattern (A) and theprovisional reference calibration pattern (B), wherein the coordinate ofthe image of the provisional reference calibration (B′) patterncorresponds to the provisional reference calibration pattern (B), or acoordinate of an image of a provisional adjustment calibration pattern(A′) based at least in part on the coordinate of the image of theprovisional reference calibration pattern (B) and the predeterminedpositional relationship between the adjustment calibration pattern (A)and the provisional reference calibration member (B), wherein thecoordinate of the image of the provisional adjustment calibrationpattern corresponds to the adjustment calibration pattern; determiningan adjusted transformation rule for the second camera by matching thecoordinate of the image of the provisional reference calibration pattern(B′) with the coordinate of the image of the provisional referencecalibration pattern (B), or by matching the coordinate of the image ofthe provisional adjustment calibration pattern (A′) with the coordinateof the image of the adjustment calibration pattern (A); and applyingprojection transform to a subsequent image captured by the second camerausing the adjusted transformation rule.